DOCOBBHT RESOHE 



BP 103 US? 



T8 00» 337 



ROTftOB 
TITLE 

INSTITOTION 

PUB DATE 
NOTE 

EDFS PRICE 
DESCFIPTORS 



lohlferd, Gerald H. 

Simplified Educational Assessoent: A Manual or 
Nontechnical School Evaluation Techniques, 
Mew York State Education Dept., Albany, Bureau of 
School Programs Evaluation. 
Oct 7«» 
67p. 

HF-$0.76 HC-$3.32 PLUS POSTAGE 
Academic Achievement; Acadeiic Records; Data 
Analysis; Data Processing; *Edttcational Assessment; 
Educational Status Conparison;, Educational 
Strategies; Elementary Education; Evaluation; 
♦Evaluation Methods; *Kanuals; *«easureBent 
Techniques; *Statistical Analysis 

ABSTRACT . , ^ ^ 

Assessment of an educational system can be done 
without using elaborate statistical techniques. Some very simple ways 
r>f analyzing data and presenting information are quite useful. This 
-anual is presented for use by those school personnel who feel a 
desire to assess the quality of their educational system, but who 
would welcome suggestions as to how the assessment might be done, it 
is designed to fill the gap between no analysis and refined 
statistical analysis. The illustrated analytical methods presented 
herein are elementary, and basic data needs are readily satisfied. 
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FOREWORD 



Evaluation is an idea wtiose time has arrived. School syscems 
can and art> presenting the public with much information on how pupils 
atv Cavinu, in various coj^nitlve arens. This pamphlet offers some 
excellent controon sense approaches to evaluation which all too often are 
neglected. 

ABSessmenc of an educational system can be done without using 
elaborate statistical technlquea. Some very simple ways of anal* ^ing 
data and presenting information are quite useful. In fact the layman 
can better understand statistics presented in chart form than th.se 
couched in statistical jargon. 

This manual springs from concerns expressed by school personnel 
to Gerald H. Wohlferd, its author, and to Charles M. Armstrong, now 
retired. Accordingly, tfie manual is presented for the use of those 
school personnel who feel a dtisire to assess the quality of their 
educational system, but who would welcome suggestions as to how the asse 
ment might be done. It .s designed to fill the gap between no analysis 
and refined statistical analysis. The illustrated analytical methods 
presented hpgt^ift are elementary^ and basic data needs are readily 
TarTstit-d. Those interested in using any of the procedures outlined 
In the text may contact the Bureau of School Programs Evaluation for 



needed help. 
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SIMPLIFIED EDUCATIONAL ASSESSMENT 



Introdu c tion 

How HOod is my scliool? This question^ which at first glance appears 
to be sensible and straightforward, is in fact very difficult to answer. No 
single answer can be given since many courses of study» grade levels, rooms 
and diverse individuals compose .1 school. Thf !Stion, then^ is really a 
series of questions whose answers, when taken together , give an indication 
of the quality of educatioa being provided. Some of the questions whose 
answer might better be sought are: Howdo the children iw my school com- 
pare with ether children in the nation, the state, amon^; themselves? How 
have Che children progressed over the years? Are the teachers equally 
effective ip teaching? 

Some of the above questions are bein^ answered every day. Others 
are avoided because of clerical costs Involved in collecting and preparing 
data for analysis, or because of iv^.norance of statistics. Fortunately, 
simple analyses are not costly, nor is a comprehensive knowledge of statls- 
tics necessary. Too, most of the data, necessary to a simple analysis, are 
already on hand or .ire easily obtained r Often all that is needed is a 
reordering, classifying, or sorting of available data. 

A sking the Rii^ht Question 

However, before educational data analysis is started a few helpful 
hints might be in order. The first is to ask only those questions which 
can 'be answered. Fpr example, the cjuestion, "How do the children In my 
school compare with other children In the nation?," Is quickly recognized 



asi being Loo broad. lt\ its present form It can nut be tinsvcrcd* One would 
want to specify at least as to svibject nrea and grade level. The question 
would better be pfuMsed, "flow do the third grade children In my school 
compare in reading skills with other children in the nation?** If desired* 
further tipeclf Icat ions can be made, so that the iiubject area is delineated 
by the particular reading skill involved, and the type of student is 
identified. The original question would thus be supplanted by multiple 
questions, such as: "How do the third grade boys in my school compare 
in reading comprehension skills with other third grade boys in the nation?" 
and» **How do the third grade boys in my school compare in reading vocabu- 
lary skills with other third grade boys across the nation?^*, etc. 

Another hint in making school evaluations is to ask questions which 
are of value--quest ions which are usable as a basis for management and 
curriculum decisions. For example, to ask how many kindergarten students 
can multiply fractions would be a waste of time. One mi^^ht better ask about 
the reading; vocabulary skills of boys and girls in order to decide whether 
more emphasis should be placed upon skill building in that subject area in 
planning subsequent curriculum activities. 

One must expect that answers to evaluation questions may reveal 
shortcoming's as well strengths. The strengths may then be capitalized 
upon» while the shortcomings may be used as points of discussion of future 
administrative policy » curricula^ organization, or program changes. 

A third hint is that evaluation is like quicksand in that the 
annwer to one question leads to asking further questions and into even 
deeper data analyses. Sooner or later a point is reached where a district 
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;nu.st dcciUo thnl it has gone fnr enough » Therefore, it might be wine to 
set up a tentcitlve evaluation program at the start, which establishes the 
limits to the evaluation. 

Accuracy of D.ita 

A caution ;!bout tftc accuracy of data is In order. No single piece 
4>f cf.iLum sfiould be accepted as an .iccurate measure until it Is least 
checked for sensibleness, and possibly duplicated at another time* Thu*^/ 
a .single I .Q. muasui t' on a clifld must only be considered accurate within 
a rnn>;e of fifteen points on cither side of the obtained score • A second 
scort- which Is .ibout the same would lend ^omv .issur.mci' th.5t both scores 
are prob.ibly riKht. 

In the same vein, bi/- increasin.^Iy more suspicious of data as it 
passes ttirou.ih several people before beln^: used in ,iny analysis. K ich 
time figure's are trmsierred from one source to anoti^er tf>e cfumce of error 
is increased r No one is infallible. Most of us have dialed a wron^» tele- 
phone numbt-r at some time. Nor is fiumin error the- sole source of poor data. 
Computers do inake errors. Usually they produce such whoppers that the 
errors are quickly seen* But dirty equipment^ old or cheap tapes, brittle 
cards^ or temperature variations, can produce di f f icult-to*-discover errors. 
Therefore, be supplcious of your data, whether it is hand copied or computer 
compiled » and cautious in making judgments from it. 

Finally, an answer to a question needs to be supported by subsequeitt 
analysis before drastic changes are made In school policy or program. One 
research finding is a hint; the second similar finding is a suggestion; the 
third is probable proof or establishes a trends 

iO 



'The foUowinH sections will show how various isducatlonal questions 

can be .inswert'd without recourse to detailftd statistical analyses or 
expensive d:itn h.jndUn^;. Most school districts conduct an achieveinent 
tfiitluK program on .1 yearly schedule. The purpose of such a testing pro- 
v',r.tm cm bf twofold. First, the tests may h.ive' pupil diagnostic capabil- 
ities. The question answered in this situation is "On what specific 
knr>wiedx:e se>',tnents do individual children require reniedlal or catch-up 
help?" Since the question .mswertd is not, in this fortn, a school evalu- 
ation question, it will not be discusst-d further at this time. Sucond, 
the test results are usually stated either in terms of averages or per- 
centiles. The question answered in this situation is, "How do my children 
(by )'rad. s in wliich tt^sLs are ^ivdn) compare in achievement (by subject 
.md HubtfSts availahle) with other childrt-n in the nation?** 

Comparison CVer Titne 

The latter analysis, i.e., a comparison to national norms, is known 
.jh .1 .status analysis, bt-cause only opt' point in time is considered. As a 
status analysiri, it has limited value since children in the school district 
seldom match the national sample on v'ltch the test norm tables are based. 
The real value Is in comparison of district scores over time. As long as 
the test battery remains the same, and the norm tables which are matched to 
it remain the same, the test norms provide a stable scale against which to 
compare one year's achiqvemunt with scores of preceding years.. District 
.ivera^es reported as grade equivalents or percentiles may be plotted on a 
chart such as Figure I. More than one test ^rea, sucii is related subtest 
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scores> may be recorded at the ^^lm^ tirae^, Hc^ny simultaneous plottings 
may, however, lead to confusion rather than clarity- 




Figure I. Third Grade Achievement Over Time 



Figure I would indicate to the district (hypothetical) for which third 
>;rnde rcidin^j^ and .arithmetic gr^ide^equivalent scores were plotted » that 
their students are achieving near the national average (3^0) as established 
by the test company* Between 1969 and 1971 arithmetic as^erages had an 
upw.ird trend. Subsequently they leveled off. Reading slumped for a number 



of years nnd then spurted upward. The administrator would want to look for 
re.isbns why the grade equivalents had changed. Were the gains in arith- 
metic and losses in reading due to a change of scheduling in I9b9 which 
assigned fifteen minutes of daily s^ond grade reading tiine to arithmetic? 
Could the gain be attributed to a successful Title I compensatory education 
program?- Was the gain in reading possibly due to a new reading curriculum 
established in. 1972? If the latter, was it because the children actually 
read better, or was the curriculum now oriented toward teaching items or 
skills tested in the reading test? Will the gain In reading scores be 
maintained? 

These and other questions will have to be answered through further 
study and experimentation. The d ange of arithmetic scores may not have 
been du;; to a scheduling change. Experimentation with an altered schedule 
In a school building, or in a single room if the district is small, could 
possibly give an answer to why the arithmetic and rca mg scores changed. 
Only time would tell if the last year's reading gain would be maintained. 
However, study of test Items In comparison with both old and new curricular 
materials may give a tentative reason for tne sudden rise In reading scores 
even before next year's test results are secured. 

As can be seen from the above discussion, the use of simple statls' 
tlcal methods can lead to more questions. When a change in score occurs, 
Che obvious response is to ask, "Why did the scores change?" A sensible 
reasoA for 'the change should be searched for. Not always are changes due to 
school-controlled situations. The real reason for the change In reading 

/ 
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scores in tht? i I ! ust r;ft ion above could have been something external from the 
school » such the starting of n summer reading program at the local library, 
an influx of able students from a local private or parochial school which 
clOficd^ ot the cumulative effect of a TV program such as Sesame Street. /Thus, 
the school administrator should look to areas both under and outside the con- 
trol of the school for possible causes of change of achievement scores. 

Subc I a s s 1 f ic a 1 1 on 

One method of searching for out-of-school causes is to divide the 
childrt*n into various subgroups and compare the achievement avcraR^es of 
thr >j^roups. One common division is by length of -time in attendance in 
tke school district. Now entrants oan depress or raise achievement 
averages. New homes can add a difCei^ent type of student depending upon 
the cost of the housing. The question then evolves to, "What type of 
sruc{<>nt ir^ causing the scores to change?" Or, "Do different student 
subi;roups achieve at diflerlnit? levels?" 

Shouici sub^roupii be found to achieve at different levels, special 
proi^ramh can be designed t^help alleviate those with depressed scores. 
For example, a district whicli covers a widci range of community types may 
wish to ji;roup the records of the children by community type, such as urban 
ceVitral, suburban, and rural frln>j;e. Should the early elementary rural 
frln;;e students have lower reading scorej, a search might be made for story- 
books more closely related to rural living. Should the urban central stu- 
dents show retarded learning skills, the administ r^tt ion may wish to place 
more emphasis upon parental participation on PTA and other school affairs. 
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Attc-ntion directed to the educational and social needs of specific sub- 
groups will, if successful, have the effect of raising the total group 
average. The above discussion points out that total group averages can 
hide the dlf ferlr ; academic levels of subgroups. 

New York State, because of Its mandated statewide testing, is one of 
the few states In which districts can answer the question, "How do the chil- 
dren in my school district compare with other children in the state?" Of 
course » ♦^he question has to be rephrased so it is more specific as to cur- 
rlcular area and grade level. The state "Pupil Evaluation Program, School 
Administrators Manual" August, 1974, offers many valuable suggestion's for 
analysing the data provided in the/ yearly report that help answer the above 
question. One addition to their suggestions could be made. On page 15 a 
chart is shown which compares over a number of years for two school buildings 
the percent o£ students achieving below the Statewide reference point. A 
school district can change the chart by omitting one school building, instead 
substituting the percent of students in the average category, and the per- 
cent of students in the above average category for the remaining school 
(see page 11 of PEP School Administrators Manual for method of securing 
percents) - 

Figure 2 illustrates how percents for students grouped by ability 
may be plotted together* It shows what can happen when attention is 
focused upon a particular group of students. In 1968 and 1969 the district 
was operating so that each group of students was achieving at about the same 
level in the subject tested. The district then became concerned with the 
Increase in the percent of students in the "below average" category, 
leachers were urged to place greater emphasis upon reducing the percent of 
students in that category. The program was successful in 1971 and 1972. 
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However, there was a simultaneous drop In the percent of students from the 
"above average" group* The trend continued in L972. Thereafter, greater 
attention was given to the loss of students ftom the "above average" group. 
Subsequently, the percent of pupils in the "above average" group rose. 
However ^ so did the percent of students In the "below average" group. 
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Figure 2. Perct-nt of Total Reading Scores Over Years 
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The above ^inalysis illustrates the need of asking an important: 
question whenever a change of policy or program has b^en placed into 



Standard OcviciL ion 

Commercial test companies also report scores In terms of standard 
deviations. A standard deviation is a calculation of the general spread 
of scores. It shows how closely the scores are grouped around the average 
(or mean). About 68 percent of the scores fall within the range of one 
r;tandard deviation above the mean to one standard deviation below the mean. 
The standard deviation changes as the spread of student scores changes, 
becoming greater as the spread of scores Increases and becoming less when 
scores ^roiip more clo.sely to the mean. Thus, comparison of standard 
deviations over the years,, roughly answers the question, "Are the students 
in a grade in succeeding years retaining the same closeness or spread of 
score.s?'* This question may be crucial when a new program or theory of 
learning Is tried. For example. Figure 3 illustrates how the standard 
deviations can be added to and subtracted from the averages and then 
plotted. The increase in the spread of scores starting In 1969 might be 
tho result of greater emphasis on allowing each child to progress at his 

4 

ofcn speed • 






Fii;uri* 3* Sprrnd oi Scores Around Averngo Over Time 



The reduction in spread of scores between 1972 and 1973 could have resulted 
£rom a policy to give special help to the slower students. Bringing their 
scores closer to the average would have the effect of reducing the standard 
deviation. 

Distributions of Scores 
' As Illustrated in some of the text, the question, "How are my 
children achieving?" can sometimes be better answered if subgroups are used 



to torn; new averages for comparison. Subgroups were chosen In Chose illus- 
trations on the basis of theory or experience. Some natural suhclassifl- 
cations come readily to mind. Sex, age, achievement leveU home background, 
and type of conununity surroundings are but a few that can be used* There 
are times when the usual classification schemes do not yield insights Into 
how adequately students aYe achieving and why- Since all^programs are not 
equally successful for all pupils, the question might then be asked, ^'Which 
students are benefiting greatly from school and which student rf. might be 
affected adversely?'' The reason for asking this question spring^ from the 
theory that a good school Is one that constantly tries to make it easier 
for Its children to learn. Those program aspects which are elffectlve can 
be Incorporated into other learning situations • Those Ineffectual should 
be discontinued. 

Frequency Distributions 

Often, identification of helpful or harmful aspects are hard to 
discover using the preceding data analysis methods. A rewarding method of 
looking at pupil scorea Is to chart them as distributions. Distributions 

of student scores which can reveal differing effects to be taking place, 
are constructed from listings of student scores, as In Table I. From 
the list of student scores, the numbet of scol-es by group is determined. 
A ch*irt of the number (frequency) of students for each score is then 
constructed. Figure 4 is an example of a frequency diagram (In this case It 
Is a line graph) constructed from the listing of scores In Table I. 
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Table 1 

List of Student Scores and Numbers in Groups 
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The frequency of scores should Ideally assume the rough shape of 
a bell. Gross Irregularltli^s from a bell shape is an indication that 
some force may be in operation which causes scores to move, to new values # 
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Achievement Score Group 
Figure 4. Frequency Distribution of Achievement Scores 

dm 

\ 

The distribution of scores in Figure 4 is bioiiK>dal, that is^ there are 
two humps* Special attention should obviously be foctised upon the chil- 
dren who compose the lower hump, and reasons for their low grouping should 
be investigated. In this example the tej^t scores could have represented a 
situation where three classrooms were combined for an audiovisual presen- 



21 



' -15- 



BESI con AVftlUIBU 



CnClon o£ the course content. * The stiuients met in the cafeterda at one end 
of which a tele vis iom set was ^lacea. When students In the lower (mode) 
were identified by name and 'ability, . there did not at first seem to be any 
sensible reason for their low scores. . A teacher In charge of the combined 
class finally saw a pattern. A large proportion of Che lower mode was 
composed of students who sat in the back of the room. It v^s then quickly 
determined that these children were too far from the television set for 
adequate viewing. Also, noise coming from the kitchen made hearing difficult. 
Furthermore,* upon further analysis of scores in the lower hump It was found 
that a few pupils v,?ere negatively affected because the surroundings were 
quite dissimilar from "the traditional classroom Isettlng. However, due to 
the general success of the audiovisual presentation, a declslcn was made 
to purchase more television sets, and to move the children back into their 
three regular classrooms where the T.V. programs could be seen and heard by 
all children. 

The foregoing discussion has pointed out how distributions of student 
scores can be used to determine probable causes of poor education. Questions 
which might be answered are: "Are the educational programs of the school 
equally effective for all children?", "If not, which children are being 
adversely affected?", "Why were these children adversely affected?", and 
"\^at changes can be made in order that adverse affects can be removed?" 

Scattergrams 

Several ways of searching for answers to the question, "How good 
is my school?", have been presented. Previous discussions have used the 
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* ^iver.i^e ^\ basis of comparison, even though children of different 
abilities compose the group for whom the average has been found* Though 
the previous analytical techniques can offer means of gaining Insights 
into school qual Ity ^. a {oost important question has been omitted. It is^ 
"Aro the children progressing at a proper speed?" Fundamental to this 
question Is the theory that a good school is also a school in which children 
.are recognized as havkig different learning speeds^ and are helped to 
proceed through the various school levels at their own unique rate. 
Central to any assessment of school quality under such a theory of education 
Is the determination of rhe correct speed at which each child should progress. 

Scattergrams offer a convenient method of determining the adequacy pf 
speed of pupil progress. They are constructed by plotting on a graph the ! 
juncture of two measures or scores for each pupil. The pattern of the plotted 
scores yields information about the progress of the group^ and the location of 
individual plotted points reveals rhe adequacy of progress of specific students. 
Two types of scat tergram plots can be done. The firsts based upon the theory 
that mental age Is a good predictor of pupil progress, plots mental age against 
academic achievement. The second, based upon the theory that past achievement 
is one of the best Indicators of present and future achievement, plots past 
achievement levels agrjinst present achievement. Both of these theories have 
been shown In past research to be true as far as educational progress is con- 
cerned, but are not necessarily true for prediction of post-griduation job 
success. In the former situation, understanding of the scattergram is aided 
if mental age Is converted into a mental grade score which is comparable in 
i^cale to :in achlf»vement grade equivalent score. A table for conversion of 
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ouintal age scores; to mcnLal grade scores; is provided In Appendix A. 

Figure 5 shows a completed plotting of student's grade 5 reading 
comprehension scores with their mental grade -scores. Each student's 
mental grade and grade equivalert scores are plotted on the scattergraro 
at the junction of their values* Each student may be entered as a tally, 
however » in Figure 5 the tallies have been totaled here for ease of viewing. 
Student A could be one of the six children having a mental grade score of 8 
and an achievement age score of 8, while student B could be the one student 
with a mental grade score of 9 and an achievement age of 12. 

A diagonal line composed of dashes has been draw% through the squares 
which have equal values on both scales. This is an expectancy line since 
children would theoretically be expected to score on achievement tests at 
a lever comparable to their mental age. Because no test is a completely 
accurate instrument, the scores cannot be expected to fall exactly along 
the line of expectancy. Errors of measurement would expand the expectancy 
band to at least one grade level above and one grade level below the dashed 
line. This band, within which scores might normally be expected to fall, 
Is outlined by tll^o light solid lines, one above and one below the dashed 
line. Tallies found in squares which are cut by, or between, the solid 
lines are to be considered as within expected ranges. T.illies of children 
found above the top solid line suggest the children to be achieving in 
academic skills above that which can be expected in light of their mental 
ages. Those tallies below the lower solid line Indicate the children are 
achieving In academic skills at a level lower than could be expected in 
relation to their mental ages. 
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Figure 5 shows many pupils to be .ichlevtng below w!uit could be 
expected. Whether the pattern shows (ichievement levels to be unexpectedly 
high or low, explanattonn should be sought. First, possible causes outside 
the school j^hould "be explored. Such things as changes in the population 
of the community or activities At the public libriiry could produce changes 
in student scores. Failure to account for such factors could lead to 
erroneous conclusions about the ei^fects of school programs. 




Ki^^urt^ 5- Scntter.gran^ of Mental Grade Scores and 
Achievement Grade for Grade 5 Reading 
Comprehension 
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A second source of possible explanations includes che character'* 
Istics of the tests and analytical procedures* If the tests do not well 
represent the curriculum* if there were irregularities in testing procor-ured^ 
or if test results are compiled so as to obscure relationships^ the results 
may be Irrelevant or biased 

Finally, factors in the school environiaehc should be 8crutini;sed 
to identify possible explanations for the performance of students. This 
is the ultimate question, for it is the school environment which can be 
chcinged to improve student achievement. 

The second type of scattergraro plots one yearns achievement against 

thdt of a subsequent year* As mentioned before, the theory behind this, 

type of evaluation is that previous achievement is related to later achieve- 

m^ent. Of course, schools try to Increase the achievement level of their 

charges until they reach their maximum* At the same time, care is taken 

not to push the students to the point of frustration. 

Paradoxically, a tally appearing above or below the expected line can 

J. 

be due to cither ^jjjouAmisual ly good score on one axis, or an unusual Ijr poor 

f' 

score- on the other axis. It is the duty of the analyst to determine if 
either of the above situations exists, or if in fact the scores are true 
and valid measures of that student *s ability or accomplishment. 

Figure 6 illustrates the plotting of reading scores of third grade 
children against the reading scores of the same children in grade four* 
Once again a diagonal dashed line has been drawn to pass through the 
juncture of expected scores. The solid lines outline the area of probable 
progress. 
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Grade Equivalent Scattergram 

Figure 6. Scattergram of Third Grade Achievement Plotted Against 
Fourth Grade Achievement- 
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The pattern of score>^ shown by the entries on Figure 6 suggest that the 
fourth grade teacher (if they were year-end tests) placed much emphasis 
ttpon raising the scores of low achievers. This is revealed by the greater 
number of children appearing above the lower end of the diagonal line. 
Unfortunately^ the better achievers may have suffered from a lack of 
attention as shown by the appearance of scores below the line at the upper 
end. Possibly too» the unusual shape could have been due to a change of 
instructional materials that did not adequately challenge the more able 
students. 

The cluster of students above the middle of the line should also 
cause a search for reasons* Maybe, these students lived in an area where 
one of the park supervisors had spent the summer evenings reading stories 
to area children and listening to them read. Or, possibly the children, 
who showed advancement above that expected of them, were involved in an 
experimental reading project* 

Several types of blank scattergrams and a table of scales to be used 
on the margins of grade equivalent scattergrams are provided in Appendix B» 
Percentiles, a common method of reporting pupil scores in high school > may 
also be plotted on a scattergram. A blank percentile scattergram is provided 
for duplication in Appendix B. 

M odes and Clusters 

An added value of scattergrams is that they reveal groupings .modes), 
the spread of scores, and how closely the scores group together. The total 
columns at the side and bottom are similar to thac of a line graph. Modes, 
then, can be seen not only through the numbers on the margins but also in 



2S 



the clustering of scores In the body of the tables. In the body of the 
scattergram the great^ir number of the tallies should cluster closely to the 
d.ished line. Groups of students* s- ores that suggest some forces at work 
may be positive (those above the major grouping) or negative (those below). 
In Figure 5, page 18» mental grade columns 7, 8, and 10 all have distinct 
clusters below the line. The column for mental grade 4 has a mode above 
the line. Two students are distinctly superior in achievement for achieve- 
ment age 12. Why are the scores of these students located where they are? 
Are the mental ages improperly measured for any of the mavericks? Are the 
10 students in mental grade column 8, rows 5 and 6, similar in some way? 
Are they also similar to the six and five students in columns 7 and 6? Are 
some of the eight students in column 5 also like the other groups of below 
achievers just discussed. 

All of these children should be identified by name. Then similar- 
ities among them could be searched out. The children in lower modes might 
have been those seated in the back of the cafeteria in the illustration 
given earlier, they might have missed school because of a flu epidemic, or 
their classroom seats may be near a poorly adjusted child. Too, any sag 
at the top of the distribution should inroediately warn the observer that 
the test used may not have enough questions which cover advanced currlcular 
materials. Such a test has a low ceiling and able students "top out" the 
test. A flat bottom at the lower end of the distribution may suggest a 
false bottom to the test. In the latter situation, correctly answering 
only one question can yield a score well above the actual ability level 
of the child. 



Sc.ittergrams, as can be seen, have many uses. They can describe status at 
some point in time (Figure 3) or proj^ress over a period of time (Figure 6). 

They are e^asy to construct » allow observation of two factors at the same 
time, while pointing to trouble and/or strong clusters and modes of students 
which deserve analytic attention. Through the use of color coded tallies » 
different groups of students may be followed in order to determine experi- 
mental, program, and/or administrative effects* Effort taken to construct 
scattergrams is minor in relation to their value* 

Tree Diagram 

The question, "How good is my school?^*, can also be answered through 
use of the tree diagram* The procedure involves following the progress of 
groups of students through school, determining their success at several 
points. The process may be done historically, that Is^ using past records 
to determine adequacy of present achievement, ot it can be done concurrently 
to determine progressive changes* 

The basic assumption behind tree diagrams is that a good school is 
one which keeps its children achieving, as well as they have ever done, if 
not better. Accordingly, a child, who has done above average scholastic 
work in the elementary school, should continue to do good work in junior 
and senior high. The measure of quality is the percent of students who 
continue to do at least the same level work throughout their school life, 
as opposed to those ^^?ho do not. Districts having high turnover will find 
this analytic method of limited use, as will those districts with poor 
record iih 
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The tree diagrams are constructed by first dividing the long-term 
students Into three groups according to their scores. Starting with the 
elementary students, their achievement Is judged to have been "Above 
Average," "Average," or "Below Average." Table 2 la provided to help the 
analyst determine the rating of single acores. 



Table 2 
Rating Scales 



Rating 


Conaaerclal Test Scores 


Teacher Grades 


Percentile 


Difference from Average 
Grade Equivalent 


Percentage 


Letter 


■8 


Above 
Average 


85-99 


Two grades above 
average and up 


85-100 


A,B 




Average 


40-84 


Average up to 2 grades 
above 


75-84 


C 


S 


Below 
Average 


1-39 


Up to average 


1-74 


D,E,F 


U 



Those students whose individual ratings were better than average throughout 
the elementary years would be assigned an "Above Average" rating. Mtore than 
two scores or grades below average Is cause for a "Below Average" rating. 
Students remaining, those who earned average grades, are assigned an "Average" 
rating. The number of "Above Average," "Average," or "Below Average" students 
Is then noted on the tree. Percentages of the three groups are then computed 
and entered on the tree diagram (Figure 7). The reader will note that 
the tree lies on its side. 
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Junior high achievement: for students in each of the three elementary 
groups--Above Average, Average^ Below Average--is judged in the sarae manner 
and entered with percentages on the- tree diagram (see Figire 7). Hopefully^ 
all students judged to be "Above Average" students in the elementary grades 
will receive the same rating in the junior high school years. Experience 
has shown such is seldom the case, though "Above Average" elementary students 
are rarely judged to be Selow Average" in junior high. "Average" students 
can and do go to both "Above Average" and "Below Average" ratings with the 
bulk staying "Average." Unfortunately, many "Below Average" students have 
been found to remain "Below Average." 

Each student in the nine junior high groups are then rated according 
to their senior high grades or scores. Totals are found for each of the 
27 groups. Totals may be entered on the tree diagram and percentages 
secured for each as shown on Figure 7. 

As can be readily seen, the tree diagram helps one follow the progress 
of a particular group of students. Of the 32 children judged to be "Above 
Average" in the elementary school, 28 continued with that ranking in the 
junior high school. However, four dropped to lower rankings In the junior 
high years. Of the two students who dropped from "Above Average" In the 
elementary grade to a "Below Average" ranking in the junior high, one 
stayed in the "Below Average" category In the senior. high, while one recovered 
his "Above Average" status. 

Many of the "Average" rated elementary children were Induced to 
greater achievement In the junior high and likewise Into the senior high 
school (see Figure 7). That nine of the "Average" group finally ended as 
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Klementary 



Junior High 



Sea lor High 



AAv 



32(23*9) 



/\bovt* Average 



134(100) 
Total 



28(87.5) Av 



Above Average 



BAv 



AAv 



2( 6.3) Av 



Average 



BAv 



2( 6.3) Av 



Below Average 



3Av 



AAv 



■7A(55.2) 
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12(16.2) Ay 
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JAv 
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3Av 
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Below Average 
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0( 0 ) Av 
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Figure 7. Tree Diagram 
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•'Below Average*' should be cause for concern^ Of the 28 *'Below Average" 
achieving students, 24 remained in that grouping throughout school • Three 
of the 28 students who were "Below Average'* in the elementary years ended 
as "Averaga" students in their high school years* 

Because of the gross method of separating the pupils when using the 
tree diagram method » some children who are on the borders of the groups 
can be expected to switch from one group to another • That does not mean 
that changes should be disregarded. Each drop should be cause for concern 
and a key to where to start looking for reasons why the drop occurred. 

One of the questions answered by tree dlagtam analysis is, "How 
well have the children progressed in school?" This same question can be 
answered by totaling the number of children found in each of the three 
rated categories for elementary^ junior, and senior high levels. Table 3 
has been constructed from the numbers of students shown in Figure 7. 



Table 3 
Totaling of Tree Diagram 





Elementary 


Junior High 


Swiilor Hlah 


Rating 


N 


% 


N 


% 


N 


7c 


Above Average 


32 


23.9 


40 


29.9 


40 


29.9 


Average 


74 


35.2 


58 


43.3 


59 


44.0 


Below Average 


28 


20.9 


36 


26.9 


35 


26,1 


Totals 


134 


100 


134 


100. 1 


134 


100 



28- 



/ 



Comparison of the-^rcenCage figures among the various levels shows 
that this school district was able to increase Che number of students in 
the "Above Average" category, but unfortunately found the percent of stu- 
dents in the "Below Average" category to Increase, also. The greatest 
change in numbers, and therefore percent s, occurred between the elementary 
and junior high levels. Though this large change could have been due to 
score shifts of those pupils near the boundaries of the categories, it 
could be due to other reasons as well. Different teacher grading policies, 
dissimilar tests used in the two levels, poor currlcular coordination, 
and/or overcrowding could have been a few of the reasons. In any event 
the Illustrative data tends to answer the question with a statement thAt 
the "Average" student is not progressing as well as one could wish. 

Tree diagrams can be constructed similar to Figure 7 using teacher 
grades or commercial achievement tests. The choice of measure would, depend 
upon completeness of data, or trust one has In the accuracy of the measure. 
However, enough data Is usually available in a district's files to allow 
analysis by tree diagraming using one type. 

Blank data collection charts, and blank summary tables and cards can 
be found in Appendix B and may be reproduced. They may be used in those 
situations where data and Information are recorded on several source .docu- 
ments. The data card Is especially helpful In pulling several pieces of 
information together on a single source. Further, the cards lend themselves 
to quick sorting Into the three categories Illustrated in Figure 7. Sorting 
can also be done by electronic data processing equipment. 
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Tree diagrams can also be used to answer questions dealing with sub-» 
classlfleatlons of students. For example^ tree diagrams can be used to 
study effects of integration upon the achievement of black or white students* 
In this situation a historical study could first be made of the progress of 
both categories of students previous to integration. Measures could be 
drawn from school records of first « third and sixth grade teacher grades or 
commercial achievement tests. After integration the same type of data» 
recorded at similar grade levels, can be placed upon tree diagrams. Up to 
seven years could pass before the analysis of the effect of Integration upon 
the scores of elementary school pupils could be completed ^ though Interim 
analyses could be made along the way. Judgment of the effect of Integration 
would depend upon comparison of pre-lntegration and post** Integration percen- 
tage figures at each of the three grade levels* A positive effect of Inte- 
gration would be shown by a decrease in the number and percent of pupils in 
"Below Average" categories and an Increase in "Average" and "Above Average" 
categories* 

Analyses can also be done using sex, X«Q«, parental education, 
parental occupation, or other classifications • To do so, separate tree 
diagrams are made for each division of the classification and cottiparlson 
is made as illustrated for the integration study. Space is provided on 
the forms in Appendix B for classifications other than those mentioned* 

Concurrent tree diagram analysis is accomplished by filling in the 
branches of the tree whenever data becomes available. The question changes 
from, "How have the children progressed?" to "How are the children pro- 
gressing?" This type of evaluation is helpful In following children who 
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have participated In special programs. Their progress can be compared with 
those not Included. Three years Is the sensible limit for tree diagram 
analysis. Above three years, the data must be grouped and averaged by a 
larger segment of tLmi, as in Figure 7. 

Modules 

For the roost part the types of analyses given in the previous 
portions of this document are generally useful to those districts having 
their students In traditional graded organizations. Indeed, one of the 
advantages of the graded system Is the tried and true measurement methods 
and analytical techniques available as aids to teachers and administrators. 
However, such is not the situation for districts using Individualized 
Instructional methods. One of the sharpest critic? .ims leveled at schools, 
which are organized so that students may advance at their own rate, Is that 
they cannot answer the question, "How are the children progressing?" 
Involved in this question is consideration of proper speed of progress. 

The traditional graded system of organization bases evaluation upon 
the progress of the group. A graded group Is usually composed of children 
of the same age. Children are scored according to their position relative 
to the group. A fast learner is rated as A, excellent, or 90, while the 
slower child, even though he works hard, earns D, poor, unsatisfactory, or 
63. Under this management system some children will always be "Above Average 
and some will always be "Below Average." Obviously, an evaluation system 
is' needed which adequately considers the unique ability of the child, when 
assignments are passed out and marks are given. 
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Expression of pupil progress in terms of moduI^B^ or units of 
curriculum^ offers evaluation possibilities for schools organized on the 
individual iJ^ed instruction system, whereas, scattergrams and tree diagrams 
offer evaluation mtjthods for schools Organized on the graded system. 

Monitoring Pupil Progress 

Modules are single units of study in ^ curricular sequence. If 
children are allowed to progress at their own rate in harmony with their 
own potential^ individuals in any group of children will be working on 
many different modules. The logistics of merely keeping track of the 
progress of children becomes difficult. A simple and sensible solution is 
to divide the curriculum into short segments — modules — and then to construct 
a chart which has nodules sequenced across the top and children's names 
listed down the side. Then, each time a child completes a inoduLe a check 
mark can be entered under the correct module for that child* Table 4 shows 
how this might be done for a single age or grade grouping. It could repre- 
sent a classroom after two months of the school year have passed e Students 
4 and 10 are slow learners, having fallen behind the main body of their 
fellows. Some are faster learners (number 7, 19 and 25) having pulled ahead* 

Estimating Pupil Progress 

Though Table 4 gives the teacher and administrator a satisfactory re- 
port of individual pupil status, it does not show whether the pupil's ppsltion 
is good or poor. Some estimate of the potential of each pupil is necessary. 
The simplest individual estimate can be made from a child's mental age or 
1,Q. Of course, each estimate should be mudified by teacher Judgment and/or 
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review of .e-ich student record file* Estimates of Individual pupil 
expected progress can be ents^red on a chart. Such charts aid the teacher 
In planning the curriculum for the coming year* Table 5 has been constructed 
to show the modules which children of a classroom might be expected to com- 
plete dur'ing the school year. The X^s represent expected mastery. For 
example, students 4 and 10 are projected to progress at a slower pace and 
will not complete as many modules as other stifdents. On the other hand^ 
students 2. 7, 14 » 19 and 25 are expected to complete more modules than the 
bulk of students. Student 16, who has fallen behind (possible due to illness) 
is expected to progress a normal ten modules during the year. 

Appendix C contains illustrative charts of mastery for children of 
differing mental ages or I,Q» levels. In these tables the mastery of modules 
is seen to Increase as the mental age or I#Q, rises. The tables In Appendix C 
are Illustrative of a modular system of 10 equal units each year^ though the 
decimals may also represent parts of the whole module that should be earned; 
Individualized instructional units set up by districts need not have regu- 
larly spaced modules during the year, nor contain the same number of modules 
each year* 

Mastery of Expecta nc y 

Neither of the above tables are, alone, sufficient for judging the 
adequacy of progress of individual pupils. The final step in converting the 
previous progress module and expectancy tables into an evaluation tool is to 
merge the two. Table 6 is such a tool. It combines expectancies as X*s and 
actual pupil progress as circles around the X's. Pupil progress is judged 
by determining how far the children have advanced toward completion of the 
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projected year's work. Gea(>raUy, the itudents whose progress has been 
charted on Table 6, have advanced the two modules as expected. Those 
students who learn more slowly have advanced only one module. Faster 
learners have advanced more than two modules, with the exception of 
student 7. The reason student 7 is slowing down should be sought. Is it 
because of absence due to illness? Could family problems be interfering 
with school study? In contrast students 14 and 25 have exceeded what 
might have been expected. Have their projected achievements been misjudged? 
Is student 14 a late bloomer? These and other questions flow naturally 
from scanning the table. 

Evaluation of total school quality is readily determined by summar- 
izing each teacher's tables on a form suth as Table 7. Few students should 
be found in either the "Above Expected" and "Below Expected" categories. 
Those that are found in either should be immediately studied. Study of 
"Above Expected" children may yield clues to program, curricular content, 
or methodologies that can be usfd to further the learning of other children. 
It may also reveal unhealthy pressures for higher 4.chievement . Of course, 
those found "Below Expected" should be cause lor concern. 

Expectancy progress tables therefore perform a four-fold function. 
* First, they are a control device whereby teachers are kept continually 
aware of student abilities and progress. Second, they t:e alarms which 
signal unusual academic behavior. Third, they are evaluation devices 
which provide to both teachers and administrators a method of measuring 
their effectiveness. Finally, use of expectancy progress tables helps 
answer the question, "How are my children progressing?" 
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Recap 

Several methods of measuring school effectiveness have been offered. 
Most involve the use of simple arithmetic • Often numbers are placed in 
chart, graphic or tabular form. Analysis is highly visual. Because the 
procedures to be followed are simple, they are easily done. Since the 
method of presentation is for-the-most-part visual, the results of the 
evaluation are readily understood. The evaluation techniques presented 
are thus useful to those not interested in a robust, mathematically 
oriented, statistical analysis. Their strength and their weakness lie in 
their simplicity. 
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wsttmamtm 

append:^ a 

MENTAL AGE TO MENTAL GRADE CONVERSION TABLE 

Derivation of Mental grade scores: 

Mental ages are usually expressed in nranths. /Mental age Is 
the dividend of the l.Q. formula: mental age t chronological age « 
I .Q. "7. To convert the figure to years, it Is di> ided by 12. Then 
so it will correspond to the grade for which it normally compares, 
5 (five years from birth to beginning of first grade) Is subtracted 
from it. Thus, a mental age of 107, divided by 12 and minus 5 equals 
a mental grade of 3.9 
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APPENDIX B 



SCATTERGRAM SCALE TABLES AND BLANK FORMS 
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GRADE EQUIVALENT SCALESXFOR SCATTERGRAM 
ROWS* AND COLUMNS** 
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*Row headings are always tor tne Kj-gnci -"^ — 

scile on the le£t running from low values at the bottom to 
greater values at the top of the scattergram. 

**Column headings are always for the lower of the two grades with 
the scale on the top of the scattergram running from left as 
smaller nuirfjers to the right in Increasingly greater values. 

- Scattergrams need not cover closely sequenced grades. Grade 3 
versus grade 6, grade 4 versus grade 6, and grade 6 versus 
grade 9 s.:o also viable uses. 
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Scattergram of Mental Grade Scores versus Achievement Grade 
Scores* (Place scales on left and top so same figure 
junctures on dashed line*) 
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Grade Equivalent Scattergraa* 

(Place scales on left and top so the grade which Is the basis for each 
scale junctures in the square • See page 20* ) 
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APPENDIX C 
FORMS FOR TREE DIAGRA>I ANALYSIS 



Contains: Collection Sheets, Synthesis Cards and Forms, 
and Several Tree Diagram Forms 
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APPENDIX D 
EXPECTATION CHARTS 
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